Document node allocatable enhancements #4532
Conversation
| @@ -73,9 +73,18 @@ introduction of allocatable resources. | |||
| An allocated amount of a resource is computed based on the following formula: | |||
|
|
|||
| ---- | |||
| [Allocatable] = [Node Capacity] - [kube-reserved] - [system-reserved] | |||
| [Allocatable] = [Node Capacity] - [kube-reserved] - [system-reserved] - Hard-Eviction-Thresholds] | |||
| ---- | ||
|
|
||
| [NOTE] | ||
| ==== | ||
| The reduction of `Hard-Eviction-Thresholds` from allocatable is a change in behavior to improve |
| [[node-enforcement]] | ||
| == Node enforcement | ||
|
|
||
| The node is able to enforce the total amout of compute resources that pods |
|
|
||
| The node is able to enforce the total amout of compute resources that pods | ||
| may consume based on the configured allocatable value. This feature significantly | ||
| improves the reliability of the node by preventing end-user pods from starving |
There was a problem hiding this comment.
"end-user" as opposed to.. what? if pods are pods, i'd delete "end-user".
There was a problem hiding this comment.
in the future, we may let special users add pods to the "system" and "kube-reserved" cgroups. in that context "end-user" meant not those, but in this context, i agree its confusing.
| may consume based on the configured allocatable value. This feature significantly | ||
| improves the reliability of the node by preventing end-user pods from starving | ||
| system services (i.e. container runtime, node agent, etc.) of required compute | ||
| resources. It is strongly encouraged that administrators set aside system |
There was a problem hiding this comment.
s/of required compute resources/for resources/
s/set aside/reserve/
There was a problem hiding this comment.
Avoid i.e and etc.
"(for example: container runtime, node agent, and so on)
| been reclaimed. To avoid (or reduce the probability of) system OOMs the node | ||
| provides xref:../admin_guide/out_of_resource_handling.adoc[Out Of Resource Handling]. | ||
|
|
||
| By reserving some memory via the `*--eviction-hard*` flag, the node attempts to evict |
There was a problem hiding this comment.
There is a mixing of "setting", "flag", and "argument" terminology. We might want to unify this.
There was a problem hiding this comment.
s/*--eviction-hard*/--eviction-hard
| and system components use up all their reservation, the memory available for pods is `28.9Gi`, | ||
| and kubelet will evict pods when it exceeds this usage. | ||
|
|
||
| If we enforce node allocatable (`28.9Gi`) via top level cgroups, then pods can never exceeds `28.9Gi` |
| If we enforce node allocatable (`28.9Gi`) via top level cgroups, then pods can never exceeds `28.9Gi` | ||
| in which case evictions would not be performed unless kernel memory consumption is above `100Mi`. | ||
|
|
||
| In order to support evictions and avoid memcg OOM kills for pods, the node sets the top level cgroup |
There was a problem hiding this comment.
s/memcg OOM killls for pods/pods being OOM killed inside the kubepods cgroup/
explaining the kubepods cgroup could be worth it so we don't have to use different, confusing language each time we are referring to it. "pod-level cgroup" already means something else so...
There was a problem hiding this comment.
i want to avoid describing the cgroup hierarchy too much here to be honest.
| is the same for user pods. | ||
|
|
||
| With the above example, node will evict pods whenever pods consume more than `28.9Gi` which will be | ||
| `<100Mi` from `29Gi` which will be the memory limits on the Node Allocatable cgroup. |
There was a problem hiding this comment.
"Node Allocatable cgroup" == kubepods? I find this sentence confusing.
| and kubelet will evict pods when it exceeds this usage. | ||
|
|
||
| If we enforce node allocatable (`28.9Gi`) via top level cgroups, then pods can never exceeds `28.9Gi` | ||
| in which case evictions would not be performed unless kernel memory consumption is above `100Mi`. |
There was a problem hiding this comment.
So... I think this is made with assumption we are using all the node-allocatable enforcement groups; pods, system-reserved, and kube-reserved.
| - "pods" <3> | ||
| ---- | ||
| <1> Enable or disable the new cgroup hierarchy managed by the node. Any change | ||
| of this setting requires a full drain of the node. It is strongly encouraged |
There was a problem hiding this comment.
s/It is strongly encouraged that users not change this value."/We recommend that users do not change this value.
| enforce node allocatable. | ||
| <2> The cgroup driver used by the node when managing cgroup hierarchies. This | ||
| value must match the driver associated with the container runtime. Valid values | ||
| are "systemd" and "cgroupfs". The default is "systemd". |
There was a problem hiding this comment.
s/"systemd"/systemd
s/"cgroupfs"/cgroupfs
| corresponding --kube-reserved-cgroup or --system-reserved-cgroup needs to be provided. | ||
| In future releases, the node and container runtime will be packaged in a common cgroup | ||
| separate from `system.slice`. Until that time, it is not encouraged for users to | ||
| change the default value of enforce-node-allocatable flag. |
There was a problem hiding this comment.
s/it is not encouraged for users to change/we recommend not changing
| separate from `system.slice`. Until that time, it is not encouraged for users to | ||
| change the default value of enforce-node-allocatable flag. | ||
|
|
||
| System daemons are expected to be treated similar to Guaranteed pods. System daemons |
There was a problem hiding this comment.
Who is expecting them to be treated similar?
"{product-title} expects system daemons to be treated similar..." ??
"System daemons should be treated similar..." ??
| exhaustively to come up with precise estimates and are confident in their ability to | ||
| recover if any process in that group is oom_killed. | ||
|
|
||
| As a result, it is strongly recommended that users only enforce node allocatable for |
There was a problem hiding this comment.
s/it is strongly recommended/we strongly recommend
|
|
||
| By reserving some memory via the `*--eviction-hard*` flag, the node attempts to evict | ||
| pods whenever memory availability on the node drops below the reserved value. | ||
| Hypothetically, if system daemons did not exist on a node, pods cannot use more than |
There was a problem hiding this comment.
"Hypothetically"?? Do we not know for certain?
s/did not/do not
| In order to support evictions and avoid memcg OOM kills for pods, the node sets the top level cgroup | ||
| limits for all pods to be `Node Allocatable + Eviction Hard Thresholds`. | ||
|
|
||
| However, the scheduler is not expected to scheduler more than `28.9Gi` and so `Node Allocatable` on node |
There was a problem hiding this comment.
s/to scheduler/to schedule
|
@derekwaynecarr Some style comments. |
|
@sjenning @mburke5678 -- made some updates, ptal |
| [[node-enforcement]] | ||
| == Node enforcement | ||
|
|
||
| The node is able to limit the total amount of compute resources that pods |
|
|
||
| Administrators should treat system daemons similar to Guaranteed pods. System daemons | ||
| can burst within their bounding control groups and this behavior needs to be managed | ||
| as part of cluster deployments. Enforcing system-reserved reservations |
| provides xref:../admin_guide/out_of_resource_handling.adoc[Out Of Resource Handling]. | ||
|
|
||
| By reserving some memory via the `--eviction-hard` flag, the node attempts to evict | ||
| pods whenever memory availability on the node drops below the reserved value. |
There was a problem hiding this comment.
s/reserved value/certain value or percentage/
| By reserving some memory via the `--eviction-hard` flag, the node attempts to evict | ||
| pods whenever memory availability on the node drops below the reserved value. | ||
| If system daemons did not exist on a node, pods are limited to the memory | ||
| `capacity - eviction-hard`. For this reason, resources reserved for evictions are not |
There was a problem hiding this comment.
s/reserved for evictions/set aside as a buffer for eviction before reaching out of memory conditions/
|
|
||
| As a result, we strongly recommended that users only enforce node allocatable for | ||
| `pods` by default, and set aside appropriate reservations for system daemons to maintain | ||
| overall node reliability. |
There was a problem hiding this comment.
Does this mean, it's recommended likes this?
kubeletArguments:
cgroups-per-qos:
- "true"
cgroup-driver:
- "systemd"
enforce-node-allocatable:
- "pods"
kube-reserved:
- "cpu=200m,memory=30G"
system-reserved:
- "cpu=200m,memory=30G"
eviction-hard:
- "memory.available<1Gi"
There was a problem hiding this comment.
@qwang1 - for the described scenario, this would be the example configuration.
we explicitly avoid defining a recommendation for the reservation at this time as its a factor of pod density.
kubeletArguments:
cgroups-per-qos:
- "true"
cgroup-driver:
- "systemd"
enforce-node-allocatable:
- "pods"
kube-reserved:
- "memory=2GI"
system-reserved:
- "cpu=1,memory=1Gi"
eviction-hard:
- "memory.available<100Mi"
|
@sjenning - updated. |
|
@derekwaynecarr Is this ready for merge? |
|
this is ready to merge. this doc should cover https://trello.com/c/QPRTj0uW/524-document-node-need-guarantees-on-os-cgroup-reservation-ops-rfe , @sjenning should write a follow-up to document the alpha support for https://trello.com/c/wq1qcjDa/455-document-node-enforce-qos-level-memory-limits which is a new flag |
this is a major feature in OCP 3.6 for node reliability.
any improvements to readability and or clarity are appreciated.
/cc @mburke5678 @sjenning @eparis